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Abstract 

This paper explores the simulation and consideration of different image-projection 
strategies for the Holodeck, a dome that will be used for highly immersive telepresence 
operations in future endeavors of the National Aeronautics and Space Administration 
(NASA). Its visualization system will include a full 360 degree projection onto the dome’s 
interior walls in order to display video streams from both simulations and recorded video. 

Because humans innately trust their vision to precisely report their surroundings, the 
Holodeck’s visualization system is crucial to its realism. This system will be rigged with an 
integrated hardware and software infrastructure — namely, a system of projectors that will 
relay with a Graphics Processing Unit (GPU) and computer to both project images onto the 
dome and correct warping in those projections in real-time. Using both Computer-Aided 
Design (CAD) and ray-tracing software, virtual models of various dome/projector 
geometries were created and simulated via tracking and analysis of virtual light sources, 
leading to the selection of two possible configurations for installation. 

Research into image warping and the generation of dome-ready video content was 
also conducted, including generation of fisheye images, distortion correction, and the 
generation of a reliable content-generation pipeline. 


Nomenclature 


CAD = Computer- Aid Design 

DOUG = Dynamic On-board Ubiquitous Graphics 

EDGE = Engineering Doug Graphics for Exploration 

F.F = Flight Deck of the Future 

GPU = Graphics Processing Unit 

iPAS = Integrated Power, Avionics, and Software 

NTPSM = Normal-Throw Projector and Spherical Mirror 

STP = Short-Throw Projector 


I. Introduction 

T he Holodeck is a confined telepresence dome — a twelve-foot-diameter hollow fiberglass sphere that can be 
entered through a latch-style door on its side. In this dome NASA’s Flight Deck of the Future (F.F) crew aims 
to create a unique telepresence experience for both ground crew and astronauts. Telepresence is simply the notion of 
fabricating the experience of being “elsewhere,” of existing and interacting realistically with a manipulable virtual 
environment. Through appeals to visual, auditory, tactile , and even olfactory technologies, the Holodeck will be 
able to provide a level of immersion that is both groundbreaking and unprecedented. Each of these sensory 
technologies will be developed independently and integrated into the dome as they are completed — ideas include 
three-dimensional surround sound, gesture and voice recognition, haptic feedback from wearable technology, and 
emitted smells. This paper, however, will focus solely on the ideation and development of the dome’s visualization 
system. 
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The choice of the dome shape was chosen particularly to 
enhance the visual aspect of telepresence. It is generally accepted in 
both the gaming industry and the virtual reality community that the 
inclusion of peripheral vision leads to a heightened sense of 
immersion in any simulation. 1 Existing visualization systems that 
engage peripheral vision include planetariums and the iDome 
system of specialist Paul Bourke, from whom many of this paper’s 
sources are drawn. 1 Still, these systems utilize hemisphere-based 
systems, which engage a visual range of only 180 degrees. Though 
this range is acceptable for a stationary subject, anyone who is free 
to rotate demands a more complete virtual environment. 

The Holodeck will be just that — a complete dome, two 
hemispheres, which together create a 360 degree range of vision all 
around the subject. Enclosed in such a full dome, a user will be able 
to freely rotate and simply see different views of a complete virtual 
environment. This, combined with other sensory technologies, 
create a very powerful telepresence experience that NASA can use 
for tele-operation of ground crews, mission planning, and 
training programs for astronauts , as well as recreation to 
improve crew morale. 

The Holodeck’ s superior immersion comes with the cost 
of a necessarily more complex projection technique. Since 
the projected images must cover the entire inner surface, 
there is no “blind spot” at which all the projection 
equipment can be harmlessly placed; indeed, any possibly 
location within the dome is a possible obstruction for light 
travelling across it. Light also cannot be shot from wall to 
wall, as the user standing inside will also throw large 
shadows, ruining the immersion. 

Because of these added complications, a standard 
equidistant-fisheye-lens projection will not work. A 
Normal-Throw Projector — that is, a standard image 
projector — will not provide adequate dome coverage on its 
own. Also, since multiple projectors would be necessary to 
cover 360 degrees, a single image would have to be divided 
onto the projectors with the edges of each projection 
blended together in a convincing way. Thus, strategies were 
chosen that: 

1 . Maximized dome coverage 

2. Minimized shadows 

3. Minimized overlap to ease edge blending 


Figure 1. The iDome. Paul Bourke ’s 
standing hemispherical system, using a 
single projector for gaming and education 
visualizations. 


Figure 2. The Holodeck. The double-hinge- 
style door is visible, as is its magnitude and 
composition. Technically considered a confined 
space until the ventilation system is installed, it 
has the capacity to fit two or three adults 
comfortably. The floor is modular, and the whole 
dome has been placed on an 80/20-style raised 
floor and added supports. 


With these considerations in mind, two strategies were chosen to pursue. 

1. Normal-Throw Projector and Spherical Mirror (NTPSM) 2 

This system implements a normal-throw 
projector and widens the image range by 
bouncing the light off of a section of spherical 
mirror that is placed in front of it. The light 
reflects, and different rays reflect at different 
angles off of the curvature of the mirror, as 
explained in Fig. 3. This magnifies the projected 
images such that a single projector can fill a 
hemisphere, if the mirror is placed at the 
hemisphere’s center However, the dome’s 

Figure 3. Top and side view of an STPSM system. 2 
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effective two hemispheres will have the user standing at the origin, necessitating a more creative solution and more 
projectors. 

2. Short-Throw Projectors (STP) 


This system implements a special type of projector — 
the STP. An STP is a projector with a low throw ratio; as 
explained in Fig. 4, a projector’s throw ratio is given by 


Where TR is the throw ratio, D is the distance from the 
projector to the screen or surface, and W is the width 
(non-diagonal) of the projected image. The higher the 
value of TR , the further away from the surface the 
projector must be to throw the same-size image. Most 
projectors have a small range of throw ratios that can be 
adjusted. Projectors with very low throw ratios (usually 
between 0.38 and 0.75) are considered STPs — these 
projectors are commonly used in presentation settings 
where, mounted from the ceiling, the can cast an image 
onto a screen while behind the presenter, removing any 
worry of shadows cast by the presenter’s gesticulation. 

Such a projection system can also be invaluable in 
settings like the Holodeck, where coverage is crucial but 
space is limited. 



Figure 4. Throw Ratio. A projector's throw ratio is 
given by the projector-screen distance D divided by 
the image width W. Projectors with lower throw 
ratios can “throw, ” or project, a larger image from 
the same distance. 


II. Simulation Technique 

Since the simulations were to be of a visual system and the most important considerations of the simulations 
were to be dome coverage, shadows thrown by users, and projector overlap, it was decided that a very visual 
representation of simulation data would be the most intuitive information to use to decide feasible projection 
strategies. The simulations would be 3D models of the dome/projector system while the projected light would be 
viewed as colored and traced rays. Ray tracing is a computational technique of tracing virtual rays on their paths 
through a fabricated 3D scene as they reflect and are absorbed by objects with differently specified material 
properties. 3 Typically this is used for high-quality image generation by placing a virtual image plane at any arbitrary 
location in the scene, but it can be used for simulation as well. 

After much research the software TracePro 4 was selected to perform the simulation. TracePro is a software that 
can generate hundreds of thousands of virtual rays, colored by frequency or by light source, as they travel through a 
scene full of objects that can be important from a CAD software such as Pro/Engineer. 5 Once generated, the surfaces 
of each object can be analyzed with different visual tools, including illuminance maps and Candela plots, which 
analyze the intensity of light as a function of 3 -dimensional position across the surface. 

Using imported CAD models along with TracePro ’s support for customizable, optics-accurate lenses, any type of 
projector ’s thrown image can be accurately modeled if its image specifications are known. It is in this manner that 
both the NTP and STP can be modeled, as shown in Fig 5. Along with a to-scale representation of the dome (Fig. 6) 
and the spherical mirror (Fig. 7), the entire dome/projector system can be modeled accurately. Ray-tracing 
simulations were conducted using two types of projectors: an NTP (Optoma HD25-FV) and an STP (Mitsubishi 
WED390U-EST) whose relevant specifications are contained in Fig. 8. 
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Figure 5. Simulated projectors. Shown above are the TracePro-simulated projectors for both a) the NTP, 
an Optoma HD25-LV, and b) the STP, a Mitsubishi WED390U-EST. Note the different spread of the 
projected beams due to the different throw ratios of each projector. 



Figure 6. CAD model of the dome. The dome has a 12 ’ diameter and is 1.5” thick. 



Figure 7. CAD model of the mirror. The spherical mirror is a quarter- section of a 9”-diameter sphere. 



Throw Ratio 

Aspect Ratio 

Dim. (W x H x D) 

NTP 

1.5 

16:9 

12.7" x 3.8" x 9.2" 

STP 

0.378 

16:10 

12.8" x 4.5" x 10.2" 


Figure 8. Projector Specifications. The NTP and STP specifications for simulation-relevant fields such as 
throw ratio, aspect ratio, and projector dimensions. 
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Using the information from Figs. 5-8, accurate 
virtual representations of various dome/projector 
geometries were explored and simulated. Simulations 
were conducted using light sources containing rays in 
a rectangular grid of 400x400 rays, or 160,000 
simulated rays per projector. A rudimentary 
representation of a 5’ 9” user (a user for which the 
dome was optimized) was constructed from primitive 
solids and placed in the center of the dome to 
accurately cast shadows from any thrown projector. 

III. NTPSM Simulations 

The simulations for the Holodeck’s visualization 
system will be explored categorically, first for the 
NTPSM system and then for the STP system. In order 
to establish a logical progression of ideas and 
advancements for feasible geometries, several 
discarded possibilities will be shown to discuss their 
flaws and set the background for their descendants. 

1. NTPSM Simulation Series 1: Pivoting 



Figure 9. Simulation Setup. This image displays the 
system of dome, projector(s), and user all in a virtual 
environment. 


Figure 10. Top view of the first STPSM 
attempt with four projectors. 


The NTPSM system was explored before the notion of the STP 
system was conceived. The initial idea was to have three or four 
projectors and the same number of mirrors, pivoted about 90 
degrees from the center. In this way the images could be thrown 
onto the dome without interfering too much with the user inside. 
The image would hit the dome’s inner surface at varying path 
lengths, providing a sloping contact. 

Fig. 1 1 shows the data from the pivoted-style NTPSM trials. 
The beam from each projector is represented by rays of a different 
color. An unexpected artifact from pivoting the NTPSM was that 
distortion from both the mirror and the differing path lengths 
rendered the image into extreme curved and spiked shapes. This 
would render an accurate post-processing such as edge-blending 
between projectors and distortion correction of those images 
extremely difficult to perform. A system with three projectors gave 



the most distortion, and the attempts to correct with a fourth projector rendered the coverage inadequate. 
It was decided that a radially symmetric image would be the best option for later post-processing. 



Figure 11. Discarded NTPSM attempts. The notion of placing pivoted projector /mirror systems in the dome was 
ruled unacceptable due to a) large semicircular shadows cast by the user, b) extreme curvature and distortion of the 
rectangular projection, and c) general image sloping and inadequate coverage. 
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2. NTPSM Simulation Series 2: Radial Symmetry 

Once it was agreed that a NTPSM system with radial symmetry was the best option, the remaining questions 
were 

1. How few projectors could be used and still retain full dome coverage? 

2. How could the mirrors be arranged to prevent shadows cast by the user? 

It was determined that by radially arranging the projectors below the floor of the dome, pointed directly in 
toward the center, and then placing mirrors radially along a smaller radius, pointing out, the images could be 
projected upward and inward through cutouts in the floor and reflected outward before ever crossing the center of 
the dome, eliminating all shadows. This strategy was tested for various numbers of projectors. 



Figure 12. Radially symmetric NTPSM with seven projectors. The radial symmetry provided a much more 
feasible outcome than pivoting. By lowering the projectors below the floor and placing them radially outside the outer 
edge of the dome as in a), a much more even coverage can be obtained. Note how, as shown in b), the light reflects 
back away from the dome ’s center and avoids ever crossing the path of the user, preventing shadows. Part c) shows 
the much more complete overhead coverage provided by seven projectors. 


However, a system of seven projectors was not economically feasible. A new system containing only five 
projectors was subsequently modeled and simulated, leading to the results in Fig. 13. As with the seven-projector 
system, dome coverage was adequate with overlap and distortion that was not overly extreme. As shown in part c), 
there was a hole in the projected image at the top of the dome. However, we know that although peripheral vision is 
key to immersion in the Holodeck, for the purposes of the virtual reality, left-to-right periphery is much more vital 
than being able to look straight up. The five-projector NTPSM system was chosen as a feasible dome geometry. 



Figure 13. Radially symmetric NTPSM with five projectors. Part a) shows the new setup with larger mirrors. The 
overlap was plentiful but acceptable, and spherical distortion led to some image curvature, as shown in b). The 
missing hole of coverage in c) is straight up and therefore not significant. 
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IV. STP Simulations 

Since the notion of radially-symmetric, inward-pointing projectors was posited, tested, and confirmed to be 
feasible, the task of finding short-throw geometry was made much simpler. Instead of being arranged around the 
edge of the dome pointed inward, the STPs were arranged near the center of the dome, pointing out. This was tested 
at two elevations. First was a simulation run with projectors below the floor (Fig. 14). 



Figure 14. STP system from under dome. Part a) shows the projector arrangement, with one active. Parts b) and c) 
speak to the lateral and vertical coverage, respectively. Note the fairly geometric overlap and lack of image curvature 
and distortion. 


However, recall that the NTPs were able to shoot a beam through cutouts in the floor because of their relatively 
narrow beams. Since the STP have such wide, diverging beams, the necessary areas of the floor cutouts reach 
prohibitively large sizes. A more feasible approach would be to hang the projectors on a support structure on the 
inside of the dome, near the top (Fig. 15). Though the projectors appear to be intersecting the dome, they are 
actually completely contained with it (projectors were tilted to more accurately portray projector image overlap). 



Figure 15. STP system from inside dome. In this system the projectors are suspended on the inside of the dome near 
the top, as in a). The coverage is comparable if not more complete than the under-dome STP system, and overlap and 
curved distortion are also similar. 
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As shown from Fig. 14 a) and 15a), a cross-section of these projection systems reveals that they too completely 
avoid intersecting the user and throwing shadows. In either STP system a 5’9” user has a 5’-5.5’ diameter circle in 
which they can walk without obstructing any ray’s path. This is slightly larger than the NTPSM system, which gave 
a walkable circle of around 4.5’. They also provide nearly equal (slightly superior) dome coverage as well as less 
overlap and curvature. Since the above-head STP system gave equal results without requiring extremely large floor 
cutouts, it was chosen as the viable STP solution. 


V. Comparing Solutions 

Both the NTPSM and STP projection systems show full telepresence dome coverage. To choose a solution, a 
close comparison is justified. The NTPSM solution : 

• Requires no support structure, which would require drilling holes in the dome 

• Houses projectors completely outside the dome, making wiring and ventilation easier 

The STP solution : 

• Gives slightly better dome coverage 

• Is all internal, requiring no cutouts 

• Gives less image overlap and distortion — easier to warp to correct later 

• More walkable area for user without casting shadow 

• No added convolution of sphere placement/geometry/trip hazard 

While neither solution has yet been chosen, the STP solution appears to be the more viable approach. 


VI. Future Work 


1. Creating Fisheye Images 

Whatever the solution and number of projectors, as 
well as whatever the content that is going to be projected, 
the pipeline of image distortion, slicing, and blending 
demands a consistent image input — this is commonly a 
fisheye image, since they contain a whole hemisphere or 
more of image data in one image, to be expanded later. If 
the video stream input is not natively in a fisheye 
projection, one can be created simply. Cubemapping is the 
process of taking four, five, or different planes of view of 
an environment (or a “cube” of images) and inflating them 
into a fisheye shape (Fig. 16). There are many software 
packages that can perform cubemapping, such as those in 
Unity 6 , software in which many simulated dome-content 
environments may be created as well. 

Some software, such as the Fulldome 7 plugin of Adobe 
After Effects, has support for images that extend past 180 
degrees of altitude, or in other words, more than a 
hemisphere. Such software would be required for the 
telepresence dome as it is a sphere more complete than a 
hemisphere or planetarium dome — called a hyper dome — 
and any image would similarly need to also extend past a 
hemisphere’s data to cover it. 


a) 



Figure 16. Cubemapping. The top, left, right, and 
bottom faces of a cube in a) are inflated into the 
artificial fisheye image of b) using software. The 
transformation is relatively simple and 
computationally inexpensive. 
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2. Image Warping 


a) 







Original image 



Once the input is standardized to 
a fisheye image, it must be warped to 
the specific configuration of the 
dome/projector solution. Since no 
two systems on the planet are exactly 
alike, each warp must be custom. 

Software that can aid is Paul 
Bourke’s MeshMapper 8 . The 
software functions by taking in a 
mesh input, usually one created in a 
computer graphics environment such 
as OpenGL. Images are parametrized 
and each pixel is processed as a set of 
texture coordinates, which are 
preserved when the mesh is read into 
the software. Those coordinates are 
matched, and the image is effectively 
“wrapped” onto the mesh, providing 
a custom distortion; indeed, with 
MeshMapper any image projection 
can be warped into any desired 
shape. Each coordinate on the mesh 

should also contain a variable for intensity to correct for the difference in brightness due to different light-path 
lengths. All of the research concerning this software and its capabilities has been completed; the only remaining task 
is to get into OpenGL and actually create the warp meshes for the dome. 



Figure 17. MeshMapper. Part a) shows how images are processed as 
texture coordinates and wrapped to a custom-built mesh to be warped. 
Part b) shows a fisheye image being warped via mesh to a projection 
suitable to be reflected off a spherical mirror. 


3. Image splitting 



After the image is warped for the specific 
dome/projector geometry, the image must be 
chopped and fed to each of the five projectors. 

The exact methodology to perform such image 
splitting is not yet exactly known, unlike the 
image warping. However, initial research has 
shown that certain NVIDIA Graphics 
Processing Units (GPUs) can interact and feed 
graphical information to all five projectors at 
once, relaying that information from a master 
computer. 

Edge blending is more of an artistic process 
that involves tapering the edge brightness of 
each overlapping projector image in order to 
“stitch together” a single realistic, seemingly 
unified image. Luckily, the supplied intensity 
mapping of the warp meshes will make edge 
blending easily, as the programmer can 
arbitrarily set the intensity variable of each 
node to a lower value along the edges of each 
image until the desired effect is created. This, once again, is a more artistic sort of process and will involve trial and 
error until a convincing edge-blending technique is created. 


Figure 18. Image splitting. The process of splitting an image, 
as shown above, is not always uniform or geometrical. 
Processes like edge blending are more artistic than scientific. 
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4. Content Generation Pipeline 


As shown in VI subsections 1-3, 
much of the research for the video 
stream post-processing has already 
been completed. What remains is to 
configure all those pieces together 
into one unified pipeline from raw 
video input to a completed, polished 
output on the dome walls. 

Fig. 19 proposes one simple way 
to view the stream of connections 
and transformations that manipulate 
the video stream on its path to 
becoming finalized dome output. 
However, much of the technology 
has only been researched and not 
implemented, so much of the 
logistics of hardware-software 
interactions and transmitting data 
still needs to be further explored. 


Raw Video 
(fromsimulation) 



Fisheye Generation 




Mesh Warping 



Image Splitting 



Send to 
Projector 


Figure 19. Proposed pipeline for content creation. Additional research must 
be performed; this is only a cursory suggestion. 


VII. Conclusion 

While ideating the Holodeck, engineers at NASA remained unsure as to the possibility of even attaining a full 
360 degree projection system, as such ground remained unbroken and was by a wide margin more complex than 
existing dome visualization technology. The CAD modeling and ray-tracing simulation provided accurate 
representations of new and different projection strategies before any installation actually had to be done. 

Furthermore, these ray- tracing simulations showed a multitude of different options for projections, leaving the 
F.F crew with a decision regarding their strategy. They can choose to opt for the NTPSM setup, which despite slight 
disadvantages such as added distortion and additional objects to consider has more support in the telepresence 
community. The other choice is, of course, the STP setup, which boasts an easier post-processing pipeline but has 
very little support as to projection on curved surfaces. This engineer notes that if ST projections onto a curved dome 
are only slightly different than ST projections on a flat screen (as opposed to a NT projection involving the added 
element of the sphere being much more complex than its flat-screen counterpart), the additional worldwide support 
may not be necessary to achieve stunning telepresence visualizations in the Holodeck. 

According to the results of the ray-tracing analysis, the notion of a hyperdome projector-based visualization 
system is indeed feasible in multiple techniques. The confirmation of such possibility allows the NASA engineers at 
the Flight Deck of the Future to continue in their development of sensory technologies to install and integrate into 
the Holodeck. 
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